Skip to content

fix(python-uv): preserve compiled bytecode after packaging - #926

Merged
bnusunny merged 12 commits into
aws:developfrom
xujiantop-crypto:fix/python-uv-compile-bytecode
Nov 18, 2026
Merged

bnusunny merged 12 commits into
aws:developfrom
xujiantop-crypto:fix/python-uv-compile-bytecode

Conversation

@xujiantop-crypto

@xujiantop-crypto xujiantop-crypto commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available: Fixes #924

Part of #925. The build option that turns this on is added in #973, and that PR is the one linked to close the issue.

Description of changes

  • make UvConfig.compile_bytecode effective when it is set, and change its default to False: bytecode compilation is opt-in. Nothing in the workflow sets it yet; feat(python-uv): add uv_compile_bytecode build option #973 adds the uv_compile_bytecode build option for that
  • when compilation is enabled, resolve an already-installed interpreter for the target runtime with uv python find --no-project --no-python-downloads <version>
  • pin that exact interpreter for dependency installation with --python, while retaining --python-version for dependency resolution
  • always pass --no-compile-bytecode to uv pip install, so uv and project uv configuration (uv.toml / [tool.uv]) never produce timestamp-based bytecode that SAM packaging would invalidate
  • drop UV_COMPILE_BYTECODE from the environment of the uv pip install subprocess. uv rejects --no-compile-bytecode when that variable is set (the argument '--no-compile-bytecode' cannot be used with '--compile-bytecode'), so it is ignored rather than allowed to fail the build. The environment still comes from original_environ(), so the PyInstaller LD_LIBRARY_PATH handling is kept
  • after a successful install, run the target interpreter with python -m compileall -f -q --invalidation-mode unchecked-hash <target> so packaged .pyc files remain valid after ZIP timestamps are normalized
  • keep missing target interpreters and bytecode compilation failures non-fatal, while logging the exit code and available diagnostics
  • decode uv/Python subprocess output as UTF-8 with replacement on Windows so UTF-8 diagnostics do not trigger locale-decoding failures
  • cover the opt-in default, explicit enablement, missing runtime/interpreter, compile failures, command construction, missing diagnostics and the UV_COMPILE_BYTECODE environment case in unit tests

-f is intentional: it replaces any timestamp-based .pyc already present in an incremental target instead of allowing compileall to skip it.

Description of how you validated changes

On the current head (develop merged in):

  • tests/unit/workflows/python_uv and tests/functional/workflows/python_uv: 161 passed, also with UV_COMPILE_BYTECODE=1 set
  • live uv 0.13.9 with UV_COMPILE_BYTECODE=1: the install failed with the flag conflict before the env fix, and succeeds after it
  • CI: all checks pass
  • Black and ruff check aws_lambda_builders passed

Earlier, on the original change:

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@xujiantop-crypto
xujiantop-crypto requested a review from a team as a code owner September 16, 2026 15:15
@github-actions github-actions Bot added pr/external stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels Sep 16, 2026

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 587257c..c2d25c3
Files: 2
Comments: 2

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 587257c..4300971
Files: 5
Comments: 1

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 587257c..6848bc2
Files: 5
Comments: 2

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 587257c..514513d
Files: 5
Comments: 2

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 587257c..0facd03
Files: 4
Comments: 1

@xujiantop-crypto xujiantop-crypto changed the title fix(python-uv): pass compile bytecode option to uv fix(python-uv): preserve compiled bytecode after packaging Sep 16, 2026

@aws-sam-tooling-bot aws-sam-tooling-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Results

Reviewed: 587257c..34b4881
Files: 5
Comments: 3

# Use the exact interpreter found by UV instead of repeating a version request that
# could resolve differently when the install command runs.
if target_python:
args.extend(["--python", target_python])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[GENERAL] The --python pin is no longer needed and makes an optional optimization change how the install itself runs.

_compile_bytecode() invokes the discovered interpreter directly:

command = [python_executable, "-m", "compileall", "-f", "-q", "--invalidation-mode", "unchecked-hash", target_dir]

The resulting .pyc files depend only on that interpreter and the installed sources — not on which interpreter uv used. Meanwhile the installed sources are already governed by --python-version and --python-platform (both handlers always pass platform="linux" plus an architecture). So --python contributes nothing to the bytecode outcome, but it does override uv's own interpreter selection for the whole uv pip install invocation — including the interpreter used for any sdist build — replacing whatever uv would have chosen (e.g. an active virtualenv) with whatever uv python find happens to return.

Dropping the pin keeps the install behavior identical to today and confines the new code path to the compilation step:

if python_version:
   args.extend(["--python-version", python_version])

with target_python used only for _compile_bytecode().

if rc == 0 and stdout:
return stdout.strip()
diagnostic = (stderr or "").strip() or (stdout or "").strip() or "no diagnostic output"
LOG.warning("Could not locate target Python %s via uv (exit code %d): %s", python_version, rc, diagnostic)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[GENERAL] A missing target interpreter is the normal case for cross-version builds, so LOG.warning here will surface a scary message on healthy builds.

LOG.warning("Could not locate target Python %s via uv (exit code %d): %s", python_version, rc, diagnostic)

Building a python3.9 function on a host that only has 3.12 installed — or any build in an image whose interpreter differs from the runtime — hits this path, and --no-python-downloads guarantees uv will not fill the gap. Since bytecode compilation is a best-effort optimization whose absence has no effect on the produced artifact, this belongs at LOG.debug (or LOG.info), otherwise users see a warning in sam build output for a build that succeeded exactly as intended. Note this is amplified by the default in comment 1: with compilation on by default, most cross-version builds will emit it.

The same applies to the "Target Python version is unavailable" warning at line 155 — _extract_python_version() raises when there is no runtime, so a falsy python_version reaching install_requirements() only happens for direct library callers.

@bnusunny bnusunny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, unchecked-hash is the right approach. Two blockers:

Functional tests fail. 11 tests in tests/functional/workflows/python_uv/ fail with TypeError: ... expected str instance, bytes found, because FakeUv returns bytes for uv python find.

Please make bytecode opt-in. With compile_bytecode defaulting to True, every build ships .pyc files. That makes artifacts larger and builds slower, and the output varies with the build machine's interpreter. Could the default be False?

@bnusunny bnusunny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points from my last review are still open. Since 2026-09-16 the branch has only had merges from develop.

  • Functional tests still fail: 11 of 13 in tests/functional/workflows/python_uv/ fail locally, and the same unit-functional jobs fail in CI on this head. FakeUv.run_uv_command returns bytes, so find_python returns bytes and building the compile command fails with expected str instance, bytes found. Make the fake return str, at least for python find.
  • Bytecode is still on by default (inline).

This also conflicts with #887 in install_requirements and tests/unit/workflows/python_uv/test_packager.py, so whichever PR merges second will need a rebase.


# Add configuration arguments
target_python = None
if config.compile_bytecode:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UvConfig.compile_bytecode still defaults to True. On develop the flag is unused, so this change makes every build run compileall and ship .pyc files. Please change the default in UvConfig to False, update test_uv_config_defaults, and fix the DESIGN.md example.

…ests

FakeUv now returns str, like SubprocessUv, so find_python no longer
breaks the compileall command. compile_bytecode defaults to False.
Unit tests and DESIGN.md are updated, and a functional test covers the
opt-in unchecked-hash path.
bnusunny
bnusunny previously approved these changes Nov 13, 2026

@bnusunny bnusunny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are in and CI is green. Thanks.

vicheey
vicheey previously approved these changes Nov 13, 2026

@vicheey vicheey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sticking with this through the review rounds. Unchecked-hash bytecode as an opt-in is a good outcome for #924/#925.

One small thing before merge: the PR description still says the default stays True and covers "the default enabled path". Could you update it to match the opt-in default, since it'll likely become the commit message?

# Conflicts:
#	aws_lambda_builders/workflows/python_uv/packager.py
#	tests/unit/workflows/python_uv/test_packager.py
uv rejects --no-compile-bytecode when UV_COMPILE_BYTECODE is set
("cannot be used with '--compile-bytecode'"), which failed every build
in that environment. Bytecode is compiled separately with unchecked-hash
invalidation, so remove the variable for the install command only.

@bnusunny bnusunny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for carrying this through, @xujiantop-crypto. Unchecked-hash bytecode compiled with the target interpreter is the right fix for #924, and opt-in is the right default for #925.

To save a round trip, I pushed two commits to your branch:

  • 3b4ebd6 merges develop. The conflicts were test-only (test_packager.py, test_utils.py) and I kept both sides.
  • 0f20821 fixes a build failure: install_requirements always passes --no-compile-bytecode, and uv rejects that flag whenever UV_COMPILE_BYTECODE is set in the environment (the argument '--no-compile-bytecode' cannot be used with '--compile-bytecode'). The install now drops that variable from its subprocess env (built from original_environ(), so the PyInstaller LD_LIBRARY_PATH handling stays), and a new unit test covers it. Verified with real uv 0.13.9: with UV_COMPILE_BYTECODE=1 set, the build failed before and succeeds after. The python_uv unit and functional suites pass (161), also with the variable set.

Remaining items, all non-blocking:

  • The opt-in can't be reached yet. The workflow builds UvConfig from the uv_* build options (workflow.py:153), and compile_bytecode isn't one of them, so it's always False in real builds. That's fine as a first step: it turns #925's dead option into a real one that is off by default. Please say so in the description, and we'll track a uv_compile_bytecode build option as a follow-up.
  • PR description: as @vicheey noted, it still says the default stays True. It will likely become the squash commit message, so please update it to the opt-in default, and mention that --no-compile-bytecode is now always passed and UV_COMPILE_BYTECODE is ignored.


# Add configuration arguments
target_python = None
if config.compile_bytecode:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing upstream sets compile_bytecode=True yet. UvConfig is built from the uv_* build options (workflow.py:153), and they don't include it, so this branch only runs in tests today. OK for this PR; worth a line in the description.

def find_python(self, python_version: str) -> Optional[str]:
"""Find an already-installed interpreter matching the requested Python version."""
rc, stdout, stderr = self.run_uv_command(
["python", "find", "--no-project", "--no-python-downloads", python_version]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uv python find <version> checks discovered virtualenvs before managed and system interpreters. That's fine, because the magic number depends only on the minor version and .pyc is arch-independent. If you want to rule out a non-CPython interpreter, cpython@<version> is an explicit request. Not required.


def _compile_bytecode(self, python_executable: str, target_dir: str) -> None:
"""Compile reusable bytecode with the interpreter selected for the target runtime."""
command = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider -j 0 for compileall. Large dependency trees (#924 measured 977 files) compile serially otherwise.

@tomaDev

tomaDev commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

I tested this branch (0f20821) against the #925 repro with SAM CLI 1.172.1. The mechanism works: with UvConfig(compile_bytecode=True) I get 196 unchecked-hash .pyc files that still load after a zip round trip with normalized timestamps.

As @bnusunny noted, though, sam build can't enable it yet, so the repro still produces 0 .pyc. Since the description says Fixes #925, merging will auto-close that issue. Could we keep #925 open until the uv_compile_bytecode build option lands, or drop the Fixes #925 line and link the follow-up there?

@bnusunny

Copy link
Copy Markdown
Contributor

Thanks for testing it. Agreed: I changed Fixes #925 here to Part of #925. The uv_compile_bytecode build option is in #973, which now carries Fixes #925, and SAM CLI maps UvCompileBytecode to it in aws/aws-sam-cli#9449. So #925 stays open until the option lands.

@vicheey vicheey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for carrying this through. One thing is still open from the earlier rounds: the description says the default stays True and covers "the default enabled path", but the code defaults to False. Could you update it to say:

  • the default is off, and the option can't be reached from a real build until #973
  • uv pip install always gets --no-compile-bytecode, so UV_COMPILE_BYTECODE and a project's uv bytecode setting no longer apply to it

Optional: -I on the compileall call, and -- before the version in uv python find.

@bnusunny
bnusunny merged commit a60ed75 into aws:develop Nov 18, 2026
124 checks passed
bnusunny added a commit to bnusunny/aws-lambda-builders that referenced this pull request Nov 18, 2026
… unset

The builder drops UV_COMPILE_BYTECODE from `uv pip install` because uv
rejects it together with --no-compile-bytecode. Before aws#926 a truthy value
made uv write timestamp .pyc; since then it silently produced none.

Read the variable in the workflow when the build option isn't given and
turn on unchecked-hash compilation with the target Python instead. Values
are parsed like uv's boolish parser (1/true/t/yes/y/on vs
0/false/f/no/n/off, case-insensitive); anything else logs a warning and
stays off. An explicit uv_compile_bytecode: false wins. Enabling via the
variable is logged at INFO.

Also document the bytecode behaviour in DESIGN.md, fix the stale
"options fail on requirements.txt" note, and keep the existing tests
hermetic against a developer's own UV_COMPILE_BYTECODE.
bnusunny added a commit to bnusunny/aws-lambda-builders that referenced this pull request Nov 18, 2026
Adds a uv_compile_bytecode (bool) build option that turns on the
unchecked-hash bytecode compilation from aws#926. It also works for
requirements.txt builds. UV_COMPILE_BYTECODE is honored when the
option is not given; an explicit false wins.

Fixes aws#925
@bnusunny bnusunny mentioned this pull request Nov 18, 2026
1 task done
bnusunny added a commit that referenced this pull request Nov 19, 2026
Adds a uv_compile_bytecode (bool) build option that turns on the
unchecked-hash bytecode compilation from #926. It also works for
requirements.txt builds. UV_COMPILE_BYTECODE is honored when the
option is not given; an explicit false wins.

Fixes #925
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr/external stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bytecode compiled during build is always discarded at runtime: SAM packaging writes 1980 timestamps, so timestamp-based .pyc never validate

4 participants